A Quantiication of Distance-bias between Evaluation Metrics in Classiication

This paper provides a characterization of bias for evaluation metrics in classiication (e.g., Information Gain, Gini, 2 , etc.). Our characterization provides a uniform representation for all traditional evaluation metrics. Such representation leads naturally to a measure for the distance between the bias of two evaluation metrics. We give a practical value to our measure by observing if the distance between the bias of two evaluation metrics correlates with diierences in predictive accuracy when we compare two versions of the same learning algorithm that diier in the evaluation metric only. Experiments on real-world domains show how the expectations on accuracy diierences generated by the distance-bias measure correlate with actual diierences when the learning algorithm is simple (e.g., search for the best single-feature or the best single-rule). The correlation, however, weakens with more complex algorithms (e.g., learning decision trees). Our results show how interaction among learning components is a key factor to understand learning performance.